Repository navigation
Vortex spark integration is a fileformat not a tableprovider - #9658
robert3005 wants to merge 1 commit into
Conversation
137c1fe to
a0db092
Compare
Merging this PR will degrade performance by 11.56%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.8 µs | 7.6 µs | -11.56% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/sparkfileformat (4c1310f) with develop (3d12093)
Footnotes
-
359 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
a0db092 to
4994e5a
Compare
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
4994e5a to
34c8e07
Compare
Add Hadoop-backed Java IO layer for Spark Signed-off-by: Robert Kruszewski <github@robertk.io>
34c8e07 to
4c1310f
Compare
jackylee-ch
left a comment
jackylee-ch
left a comment
There was a problem hiding this comment.
Built this branch and ran the Spark 4.0 module's compat suite against 4.1.2 and 4.2.0, plus a few PySpark 4.2.0 checks; notes inline. Collated string predicates arrive as CollatedFilter, fall through the translator and stay with Spark, so #10065 does not reproduce here: name = 'alice' on a UTF8_LCASE column returns 3 rows on this branch (Spark 4.2.0) and 1 on 0.88.0 (Spark 4.1.3).
Checked with AI assistance; the numbers are from local runs.
| } | ||
|
|
||
| if (sparkLine == "4.0") { | ||
| register<JvmTestSuite>("spark41CompatTest") { |
There was a problem hiding this comment.
With spark41 set to 4.2.0 this suite passes here (the S3Mock tests skip without Docker), while 0.88.0 fails every read on 4.2.0 with NoSuchMethodError, since CatalogV2Util.structTypeToV2Columns takes (StructType, boolean) from 4.2 on. Could the compat suite cover 4.2.0 too, and should I send a small fix for the current connector in case this misses the next release?
| if (dataType instanceof FloatType) { | ||
| return numericLiteral(value, Expression.DType.F32, number -> Expression.literal(number.floatValue())); | ||
| } | ||
| if (dataType instanceof DoubleType) { | ||
| return numericLiteral(value, Expression.DType.F64, number -> Expression.literal(number.doubleValue())); | ||
| } |
There was a problem hiding this comment.
Spark's re-check can't bring back rows Vortex already dropped, and Vortex compares floats in total order (x = 0.0 returns 1 of 2 rows here, x >= 0.0 5 of 6), so should these return Optional.empty() until #10064 is resolved?
This branch was successfully deployed
2 participants
Following in the footsteps of duckdb integration we now claim to be a fileformat
instead of tableprovider thus better aligning with existing functionality
We also add shared java io layer that can you JvmWrite/JvmReadAt
Signed-off-by: Robert Kruszewski github@robertk.io